fun little detour, turns out our OSX tests in CI were never running with their intended deps (s3fs was missing) because we declared the environment circularly. This PR fixed that, and revealed that our s3 test fixtures are not getting cleaned up correctly on OSX, which raises warnings which fail our tests. This is now fixed in this PR.

fun little detour, turns out our OSX tests in CI were never running with their intended deps (s3fs was missing) because we declared the environment circularly. This PR fixed that, and revealed that our s3 test fixtures are not getting cleaned up correctly on OSX, which raises warnings which fail our tests. This is now fixed in this PR.

this is one of several times we've encountered something similar with the hatch env setup. I think we should consider switching from hatch for environment management. pixi is more explicit in its environment setup and is catching on with friends of zarr (e.g., pydata/xarray#10888).

fun little detour, turns out our OSX tests in CI were never running with their intended deps (s3fs was missing) because we declared the environment circularly. This PR fixed that, and revealed that our s3 test fixtures are not getting cleaned up correctly on OSX, which raises warnings which fail our tests. This is now fixed in this PR.

this is one of several times we've encountered something similar with the hatch env setup. I think we should consider switching from hatch for environment management. pixi is more explicit in its environment setup and is catching on with friends of zarr (e.g., pydata/xarray#10888).

Until we have non-pypi dependencies I'm not sure pixi adds much here -- i'm not sure it would prevent the issues this PR fixes

fun little detour, turns out our OSX tests in CI were never running with their intended deps (s3fs was missing) because we declared the environment circularly. This PR fixed that, and revealed that our s3 test fixtures are not getting cleaned up correctly on OSX, which raises warnings which fail our tests. This is now fixed in this PR.

this is one of several times we've encountered something similar with the hatch env setup. I think we should consider switching from hatch for environment management. pixi is more explicit in its environment setup and is catching on with friends of zarr (e.g., pydata/xarray#10888).

Until we have non-pypi dependencies I'm not sure pixi adds much here -- i'm not sure it would prevent the issues this PR fixes

You're right that the flagship feature of non-pypi deps isn't necessary for zarr-python. At the same time, the circular zarr[test] self-reference pattern is a hatch-specific footgun — pixi doesn't support that pattern at all. Dependencies are declared explicitly per feature/environment, so there's no mechanism to accidentally create a broken self-reference in the first place. This makes environment misconfiguration like the one this PR fixes much harder to introduce silently. #2872 was the other time I discovered various issues with our hatch setup.

It's still mostly a personal preference and while I'd prefer to use pixi across projects rather than hatch for just zarr-python my opinion on this is not very strong so that's fine.

That's a good point about how pixi would prevent dep declaration cycles by design. I wonder if plain uv would offer the same protection. Either way, I do think we are in a good place in this pr thanks to the new dependency group logic. If we have more trouble with hatch in the future we should definitely revisit this.